在前一天的Day 24 - Shiaoji.Login踩坑經驗及修正中,談到在執行login動作時,未考慮要等待contract fetch動作完成,導致所抓的資料內容不完整。這次的踩坑經驗,也讓我重新去思考Day 23 - 重覆呼叫shioaji.Shioaji()產生的記憶體問題中,是否也會因為相同的問題,影響測試的結果。
所以,我把當時測試的程式碼,改為以下方式:
import os, psutil, gc, time
import shioaji as sj
from dotenv import load_dotenv
load_dotenv('D:\\python\\shioaji\\.env') #讀取.env中的環境變數
process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
for i in range(5):
api = sj.Shioaji(simulation=True)
print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
start_time = time.time()
# api.login(
# person_id=os.getenv('YOUR_PERSON_ID'),
# passwd=os.getenv('YOUR_PASSWORD'),
# contracts_timeout=10000
# )
api.login(
person_id='PAPIUSER01',
passwd='2222'
)
print(api.Contracts) #增加此行,等待所有的Contract資料fetch完成後,才繼續
print(f'contract exe time:{time.time()-start_time}')
print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
api.logout()
del api
gc.collect()
print(f'mem usage after del api: {process.memory_info()[0]/float(2 ** 20)}MB')
這次的程式中,在login後增加一行print(api.Contracts),確保所有的Contract資料都fetch完成後,才繼續後續的動作;另外在記憶體計算部份,因為換算時是除以2的20次方,所以單位是MB而不是KB。而這一次的測試,也一併測試Contract fetch所需要的時間,以便與優化後做比較。
本次的測試,也依照上次的4個不同場景做測試,測試結果如下:
程式點 | windows&正式環境 | windows&測試環境 | ubuntu&正式環境 | ubuntu&測試環境 |
---|---|---|---|---|
start | 33.984375 | 33.9296875 | 34.55859375 | 34.4375 |
0-initial | 37.87890625 | 38.02734375 | 38.0234375 | 37.8046875 |
0-login | 226.46875 | 231.0390625 | 220.9765625 | 39.03515625 |
0-contract fetch time | 7.203447580337524 | 8.030523300170898 | 5.9245991706848145 | 7.924567461013794 |
1-initial | 226.4453125 | 231.01171875 | 220.9765625 | 229.625 |
1-login | 406.69921875 | 409.90625 | 219.4492188 | 400.9609375 |
1-contract fetch time | 7.261681318283081 | 8.347200632095337 | 6.103384017944336 | 8.131371021270752 |
2-initial | 406.640625 | 413.8671875 | 398.80078125 | 415.3203125 |
2-login | 587.5625 | 597.703125 | 586.6640625 | 597.3125 |
2-contract fetch time | 6.988409757614136 | 8.512909173965454 | 6.160722017288208 | 7.1584250926971436 |
3-initial | 587.51171875 | 597.69140625 | 586.6640625 | 597.3125 |
3-login | 767.63671875 | 780.59765625 | 774.5546875 | 781.94921875 |
3-contract fetch time | 7.1759161949157715 | 8.046003103256226 | 6.2177183628082275 | 7.363017797470093 |
4-initial | 767.58203125 | 780.578125 | 774.5546875 | 781.94921875 |
4-login | 950.8515625 | 963.15234375 | 954.59375 | 970.33203125 |
4-contract fetch time | 7.381143569946289 | 8.393444299697876 | 6.169715166091919 | 7.520856142044067 |
del & gc | 950.81640625 | 963.05078125 | 954.59375 | 970.33203125 |
這一次的測試結果,每個測試場景的記憶體使用量及時間,其實都差不多;而上一次的測試結果,測試環境及Linux環境下,之所以都會比windows平台或是正式環境要來得低,都是因為Contract資料下載不完整,而得到錯誤的測試結果 | ||||
雖然在測試程式中,有執行del api及gc.collect,但測試的結果卻是記憶體使用量沒有下降,因為Python的記憶體空間都是由Python memory manager在做管理,而實際上會不會清理出記憶體空間,也是要看是不是符合memory manager的管理機制,並不像C/C++可以直接在程式中做記憶體管理。 | ||||
關於Python memory manager,可參考下列文章: | ||||
Python 3.10.0 Documentation » Python/C API Reference Manual » Memory Management | ||||
Memory Management in Python | ||||
Python 用del删除变量以后为什么还是OOM(Python的内存管理与垃圾回收) |
在上次那篇有談到,記憶體使用量優化的方式,這裡就說明程式碼調整內容,以及優化後的測試結果
優化的分別程式碼如下:
import os, psutil, time
import shioaji as sj
from dotenv import load_dotenv
load_dotenv('D:\\python\\shioaji\\.env')
process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
process_start_time = time.time()
api = sj.Shioaji() #api宣告移至最外層,不重覆執行
for i in range(5):
print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
start_time = time.time()
api.login(
person_id=os.getenv('YOUR_PERSON_ID'),
passwd=os.getenv('YOUR_PASSWORD')
)
print(api.Contracts)
print(f'({i})contract fetch time:{time.time()-start_time}')
print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
api.logout()
print(f'process total exe time:{time.time()-process_start_time}')
import os, psutil, time
import shioaji as sj
from dotenv import load_dotenv
load_dotenv('D:\\python\\shioaji\\.env')
process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
process_start_time = time.time()
my_contract = None #宣告my_contract物件
for i in range(5):
api = sj.Shioaji()
print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
start_time = time.time()
# 當my_contract為None時,執行contract fetch動作
if my_contract is None:
api.login(
person_id=os.getenv('YOUR_PERSON_ID'),
passwd=os.getenv('YOUR_PASSWORD')
)
print(api.Contracts) #等待Contract fetch完成
my_contract = api.Contracts #將my_contract指向api.Contracts
else:
print('fetch_contract=False...')
api.login(
person_id=os.getenv('YOUR_PERSON_ID'),
passwd=os.getenv('YOUR_PASSWORD'),
fetch_contract=False #設為False,表示login後不執行contract fetch
)
print(my_contract)
print(f'({i})contract fetch time:{time.time()-start_time}')
print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
api.logout()
print(f'process total exe time:{time.time()-process_start_time}')
import os, psutil, time
import shioaji as sj
from dotenv import load_dotenv
load_dotenv('D:\\python\\shioaji\\.env') #讀取.env中的環境變數
process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
process_start_time = time.time()
api = sj.Shioaji() #api宣告移至最外層,不重覆執行
for i in range(5):
print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
start_time = time.time()
# 當api.Contracts為None時,執行contract fetch動作
if api.Contracts is None:
api.login(
person_id=os.getenv('YOUR_PERSON_ID'),
passwd=os.getenv('YOUR_PASSWORD')
)
else:
print('fetch_contract=False...')
api.login(
person_id=os.getenv('YOUR_PERSON_ID'),
passwd=os.getenv('YOUR_PASSWORD'),
fetch_contract=False #設為False,表示login後不執行contract fetch
)
print(api.Contracts)
print(f'({i})contract fetch time:{time.time()-start_time}')
print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
api.logout()
print(f'process total exe time:{time.time()-process_start_time}')
程式點 | sj.Shioaji()移至外層 | 僅第一次執行時,執行contract fetch並將Contracts資料儲存 | sj.Shioaji()移至外層+僅第一次fetch
------------- | ------------- | -------------
start | 34.0390625 | 33.9765625 | 33.9765625
0-initial | 37.97265625 | 37.84765625 | 38.01171875
0-login | 226.421875 | 226.765625 | 226.22265625
0-contract fetch time | 6.864650726318359 | 7.08452582359314 | 6.8395609855651855
1-initial | 226.4453125 | 226.91015625 | 226.16015625
1-login | 247.984375 | 227.140625 | 226.31640625
1-contract fetch time | 1.8755991458892822 | 1.5779919624328613 | 1.5284152030944824
2-initial | 249.35546875 | 227.18359375 | 226.265625
2-login | 250.640625 | 227.34765625 | 226.4375
2-contract fetch time | 2.2606329917907715 | 1.7178151607513428 | 1.3352644443511963
3-initial | 259.97265625 | 227.33203125 | 226.39453125
3-login | 273.60546875 | 227.5 | 226.5625
3-contract fetch time | 2.4294867515563965 | 1.4267780780792236 | 1.6401028633117676
4-initial | 274.13671875 | 227.48828125 | 226.5078125
4-login | 276.0234375 | 227.6484375 | 226.671875
4-contract fetch time | 2.1341898441314697 | 1.4912939071655273 | 1.2037618160247803
process total exe time | 16.376935720443726 | 13.961941003799438 | 12.832036018371582
可以看到,這三種優化方式,從第二次登入後的測試結果,記憶體使用量或是contract fetch time,都跟優化前的有很明顯示的改善。